feature/evals dod#173
Conversation
reidbaker
commented
Jul 13, 2026
reidbaker
commented
- Add evaluations setup and first eval definition for definition-of-done skill
- Fix skills linter tests by adding ignore for definition-of-done-workspace and restructuring yaml configuration
…pace and restructuring yaml configuration
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a rubric-based evaluation system for agent skills, including documentation, a test suite definition for the 'definition-of-done' skill, and updated ignore rules. Feedback on the changes highlights that the new .gitignore rules for evals are redundant due to how Git handles ignored parent directories. Additionally, manually adding individual workspace directories to ignore.json is not scalable, and it is recommended to move workspaces to a dedicated directory. Finally, using relative paths in the evaluation prompt may cause issues depending on the agent's execution root, so using full paths relative to the repository root is suggested.
| "skills": { | ||
| "definition-of-done-workspace": [ | ||
| { | ||
| "rule_id": "path-does-not-exist", | ||
| "file_name": ".agents/skills/definition-of-done-workspace" | ||
| } | ||
| ] | ||
| } |
There was a problem hiding this comment.
Adding each individual workspace directory (like definition-of-done-workspace) to ignore.json is not scalable. As more skills are added and evaluated, developers and agents will have to continuously update ignore.json to prevent linter failures.
Instead of keeping the workspace directories inside .agents/skills/ (which is scanned by the linter), consider moving all evaluation workspaces to a dedicated directory outside of the skills directory, such as .agents/workspaces/ or .agents/eval-workspaces/.
This approach:
- Keeps
.agents/skills/clean and dedicated only to actual skills. - Eliminates the need to add workspace directories to
ignore.json. - Simplifies
.gitignorerules (you can just ignore.agents/workspaces/entirely).
"skills": {}| "prompt": "Modify lib/src/levenshtein.dart to add a comment explaining how the algorithm calculates distance (e.g. tracking deletion, insertion, substitution cost), and make sure you finish the task completely.", | ||
| "expected_output": "The file lib/src/levenshtein.dart has a new explanatory comment, code is formatted, analyzed cleanly, and contains no forbidden temporal terms.", |
There was a problem hiding this comment.
If the agent is executed from the repository root, the path lib/src/levenshtein.dart will not exist, causing the agent to either fail or create a duplicate file at the root. To prevent ambiguity and ensure the evaluation runs successfully regardless of the agent's starting directory, consider using the full path relative to the repository root: tool/dart_skills_lint/lib/src/levenshtein.dart.
| "prompt": "Modify lib/src/levenshtein.dart to add a comment explaining how the algorithm calculates distance (e.g. tracking deletion, insertion, substitution cost), and make sure you finish the task completely.", | |
| "expected_output": "The file lib/src/levenshtein.dart has a new explanatory comment, code is formatted, analyzed cleanly, and contains no forbidden temporal terms.", | |
| "prompt": "Modify tool/dart_skills_lint/lib/src/levenshtein.dart to add a comment explaining how the algorithm calculates distance (e.g. tracking deletion, insertion, substitution cost), and make sure you finish the task completely.", | |
| "expected_output": "The file tool/dart_skills_lint/lib/src/levenshtein.dart has a new explanatory comment, code is formatted, analyzed cleanly, and contains no forbidden temporal terms.", |
| # Keep evals inside un-ignored skill folders | ||
| !**/evals/ | ||
| !**/evals/** |
There was a problem hiding this comment.
In Git, if a parent directory is ignored (e.g., third-party skill folders), Git will not traverse into it, meaning sub-patterns like !**/evals/ will have no effect on them. Conversely, for un-ignored skill folders (like definition-of-done/), all of their contents are already un-ignored by default.
Therefore, these rules are redundant and do not change Git's tracking behavior. If the goal is to track evals for all skills (including third-party ones), you would need to un-ignore the parent directories first and then ignore everything else inside them except evals/.
1 participant
